Navigating the Data Lake: Unsupervised Structure Extraction for Text-formatted Data

Many organizations routinely accumulate automatically-generated semi-structured log file datasets; these datasets remain unused and occupy wasted space—this phenomenon has been termed as the “data lake” problem. One approach to put these datasets to use is to convert them into a structured relational format, following which they can be analyzed in conjunction with other datasets. To address this, we present CATAMARAN, an automatic structure extraction tool that requires no human supervision. CATAMARAN automatically identifies field and record endpoints, separates the structured parts from the unstructured noise or formatting, and can tease apart multiple structures from within a dataset, in order to efficiently extract structured relational datasets from semi-structured log files, at scale with high accuracy. Compared to unsupervised adaptations of supervised structure extraction tools developed in prior work, CATAMARAN makes fewer assumptions and achieves much higher accuracy. In particular, CATAMARAN can successfully extract structured information from all datasets used in prior work in supervised structure extraction, and can achieve 95% extraction accuracy on automatically collected log files from GitHub.